Issue #87 create cleanup script for all s3 buckets #109

1jc · 2022-03-01T19:30:00Z

#87
The script should list the buckets and delete them after confirmation. per instructor

The goal is to make a script listing the S3 buckets and then deleting them after confirmation. Testing will made to make sure script works. 1jc

DATA ISSUE
The need script to list the s3 buckets and delete such buckets.

SCOPE

write python script for s3 bucket listing.
write python script for s3 bucket deleting.

Major Obstacles:

Instructor tested PR Thursday March 3, 2022. s3BucketList.py worked fine. s3BucketDelete.py worked but he wanted me to fix the script it where it should be automated versus original script deleting each bucket one by one making sure the bucket is empty.

Resolved Obstacle:

#116
This is a bug in the #87
The script doesn't delete all S3 buckets but only one
The script requires manual input and cannot be run from an automation machine. This should be changed to command line parameters
You can use #85 as example how to write automation
Instructor issued another issue which is issue #116 Fix the script to delete all s3 buckets even with data in connection there is a bug in issue 87 .

Project report for Instructor
Project Duration Sprint 3: started February 17, 2022, ending February 28, 2022
Coordination: DevOps Team issue assigned to 1jc
Report Date: February 28, 2022
Assumptions: N/A

1jc created script to list s3 bucket listing

1jc created script to delete a s3 bucket listing for clean up

1jc · 2022-03-01T20:56:23Z

TESTING:
Requirements:

AWS CLI configuration
Python Boto3 installed
For testing:

Need AWS CLI
$ aws –version
If your system does not have AWS CLI
$ aws configure
Need access key id, secret access key, default region name as us-west-2
Install Python Boto3 in AWS CLI
$ python -m pip install boto3
Import boto3
Client = boto3.client(
‘s3’,
aws_access_key_id = ACCESS_KEY,
aws_secret_access_key = SECRET_KEY,
aws_session_tokens = SESSION_TOKEN
)
$ git pull
How do I run a git pull command?
Now go back to the original folder and follow the instructions:

First,
run git status
Git will tell you the repository is clean.
Then,
run git fetch
Next
run git status
Git will say your branch is one commit behind
4.run git pull
updates your local branch
At command prompt navigate to the branch where script file is available in this case branch - :issue_#87_create_cleanup_script_for_all_s3_buckets
type python then the script s3BucketList.py renamed to ListS3Bucket.py
enter to run script
verify list of s3 buckets
type python then the script s3BucketDelete.py renamed to DeleteS3Bucket.py
note name of s3 bucket you want to delete – script will ask for your assistance on this

1jc · 2022-03-01T20:58:00Z

IMPLEMENTATION
Issue #87 Create a cleanup script for all s3 buckets

Research
Write python scripts
Create branch in repository
issue_#87_create_cleanup_script_for_all_s3_bucket
Create file
s3BucketList.py for listing s3 buckets
Create file
s3BucketDelete.py for deleting s3 buckets
Put files into branch made
Do a git pull (to update your local)
Testing to be done see directions for testing script
Need at least two reviewers for two approval for pull request

1jc · 2022-03-01T20:59:18Z

Activities Documentation	Date	Activity	Time Spent
Sprint 3 Issue 87	Feb 24	Research	1.0 hours
Sprint 3 Issue 87	Feb 27	Research	hours
Sprint 3 Issue 87	Feb 28	Documentation	2.0 hours
Sprint 3 Issue 87	Feb 28, Mar 1	Folder, Initiate Pull, commits	9.0 hours
Totals			12.0 hours

1jc · 2022-03-01T22:51:17Z

TESTING DONE: Buckets are empty as per script run in AWS CLI

1jc · 2022-03-01T22:58:35Z

TESTING DONE: AWS CLI run s3BucketList.py and s3BucketDelete.py

C:\Users\jcdam>aws --version
aws-cli/2.4.18 Python/3.8.8 Windows/10 exe/AMD64 prompt/off

C:\Users\jcdam>aws configure

C:\Users\jcdam>python -m pip install boto3
Collecting boto3
Downloading boto3-1.21.10-py3-none-any.whl (132 kB)
|████████████████████████████████| 132 kB 656 kB/s
Collecting botocore<1.25.0,>=1.24.10
Downloading botocore-1.24.10-py3-none-any.whl (8.6 MB)
|████████████████████████████████| 8.6 MB 33 kB/s
Collecting jmespath<1.0.0,>=0.7.1
Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting s3transfer<0.6.0,>=0.5.0
Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB)
|████████████████████████████████| 79 kB 392 kB/s
Collecting urllib3<1.27,>=1.25.4
Downloading urllib3-1.26.8-py2.py3-none-any.whl (138 kB)
|████████████████████████████████| 138 kB 3.3 MB/s
Collecting python-dateutil<3.0.0,>=2.1
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
|████████████████████████████████| 247 kB 2.2 MB/s
Collecting six>=1.5
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: urllib3, six, python-dateutil, jmespath, botocore, s3transfer, boto3
Successfully installed boto3-1.21.10 botocore-1.24.10 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.2 six-1.16.0 urllib3-1.26.8
WARNING: You are using pip version 20.2.3; however, version 22.0.3 is available.
You should consider upgrading via the 'C:\Users\jcdam\AppData\Local\Programs\Python\Python39\python.exe -m pip install --upgrade pip' command.

C:\Users\jcdam\ad440-winter2022-thursday-repo\devops>git branch -r
origin/issue_#87_create_cleanup_script_for_all_s3_buckets

C:\Users\jcdam\ad440-winter2022-thursday-repo\devops>git checkout issue_#87_create_cleanup_script_for_all_s3_buckets
Switched to a new branch 'issue_#87_create_cleanup_script_for_all_s3_buckets'
Branch 'issue_#87_create_cleanup_script_for_all_s3_buckets' set up to track remote branch 'issue_#87_create_cleanup_script_for_all_s3_buckets' from 'origin'.

C:\Users\jcdam\ad440-winter2022-thursday-repo\devops>git ls-files
DeployCloudformationS3Bucket/StackTemplate.json
DeployCloudformationS3Bucket/deployStackS3Bucket.py
DeployDynamoDB/deployDynamoDB.py
DeployDynamoDB/dynamoDBStackTemplate.json
DeployLambdaAutomation/automationScriptLambda.sh
DeployLambdaAutomation/cloudFormationLambdaTemplate.yml
DeploySagemakerNBI/deploySagemakerNBI.py
DeploySagemakerNBI/sagemakerStackTemplate.json
s3BucketDelete.py
s3BucketList.py

C:\Users\jcdam\ad440-winter2022-thursday-repo\devops>s3BucketList.py
Traceback (most recent call last):
File "C:\Users\jcdam\ad440-winter2022-thursday-repo\devops\s3BucketList.py", line 5, in
response = client.list_buckets()
File "C:\Users\jcdam\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\client.py", line 395, in _api_call
return self._make_api_call(operation_name, kwargs)
File "C:\Users\jcdam\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\client.py", line 725, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied

C:\Users\jcdam\ad440-winter2022-thursday-repo\devops>s3BucketDelete.py
Please provide bucket name to be deleted: kwargs
Before deleting the bucket we need to check if its empty. Checking...
Traceback (most recent call last):
File "C:\Users\jcdam\ad440-winter2022-thursday-repo\devops\s3BucketDelete.py", line 13, in
objs = client.list_objects_v2 (Bucket = bucket_name)
File "C:\Users\jcdam\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\client.py", line 395, in _api_call
return self._make_api_call(operation_name, kwargs)
File "C:\Users\jcdam\AppData\Local\Programs\Python\Python39\lib\site-packages\botocore\client.py", line 725, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectsV2 operation: The specified bucket does not exist

C:\Users\jcdam\ad440-winter2022-thursday-repo\devops>

1jc made changes to line 16 per request BT

nceres · 2022-03-03T06:06:22Z

When I try to run S3BucketDelete.py I get this error:

Please provide bucket name to be deleted: nwc-test-bucket
Before deleting the bucket we need to check if its empty.  Checking...
0
Traceback (most recent call last):
  File "/Users/naceres/Documents/GitHub/ad440-winter2022-thursday-repo/devops/s3BucketDelete.py", line 16, in <module>
    filecount = objs ['Keycount']
KeyError: 'Keycount'
naceres@Nates-MacBook-Pro ad440-winter2022-thursday-repo % /Users/naceres/opt/anaconda3/bin/python /Users/naceres/Documents/GitHub/ad440-winter2022-thursday-repo/devops/s3BucketDelete.py
Please provide bucket name to be deleted: nwc-test-bucket
Before deleting the bucket we need to check if its empty.  Checking...
0
Traceback (most recent call last):
  File "/Users/naceres/Documents/GitHub/ad440-winter2022-thursday-repo/devops/s3BucketDelete.py", line 16, in <module>
    filecount = objs ['Keycount']
KeyError: 'Keycount'

The other script works.

nceres · 2022-03-03T06:08:47Z

When I try to run S3BucketDelete.py I get this error:

Please provide bucket name to be deleted: nwc-test-bucket
Before deleting the bucket we need to check if its empty.  Checking...
0
Traceback (most recent call last):
  File "/Users/naceres/Documents/GitHub/ad440-winter2022-thursday-repo/devops/s3BucketDelete.py", line 16, in <module>
    filecount = objs ['Keycount']
KeyError: 'Keycount'
naceres@Nates-MacBook-Pro ad440-winter2022-thursday-repo % /Users/naceres/opt/anaconda3/bin/python /Users/naceres/Documents/GitHub/ad440-winter2022-thursday-repo/devops/s3BucketDelete.py
Please provide bucket name to be deleted: nwc-test-bucket
Before deleting the bucket we need to check if its empty.  Checking...
0
Traceback (most recent call last):
  File "/Users/naceres/Documents/GitHub/ad440-winter2022-thursday-repo/devops/s3BucketDelete.py", line 16, in <module>
    filecount = objs ['Keycount']
KeyError: 'Keycount'

The other script works.

Okay just saw you pushed some changes and it is working for me now.

nceres

Please add these scripts to their own subfolder (DeleteS3Bucket) under 'devops'. I suggest the naming convention 'deleteS3Bucket.py' to be consistent with the other folders/scripts.

1jc · 2022-03-03T06:26:04Z

@nceres Please add these scripts to their own subfolder (DeleteS3Bucket) under 'devops'. I suggest the naming convention 'deleteS3Bucket.py' to be consistent with the other folders/scripts.

I tried doing what you suggested when I first started the issue the folder system would not let me for some reason - so I put these into my own branch

@nceres Please add these scripts to their own subfolder (DeleteS3Bucket) under 'devops'. I suggest the naming convention 'deleteS3Bucket.py' to be consistent with the other folders/scripts.
I tried doing what you suggested when I first started the issue the folder system would not let me for some reason - so I put these into my own branch. For some reason when they made corrections on behalf in the beginning it is where I could not make any subfolders in the devops to deter me. @CodeSmartBiks

1jc · 2022-03-03T06:30:47Z

1jc nceres reviewed and run the two scripts stating that it works. Cannot change the subfolder criteria because system would not let me. I made the necessary corrections for the s3BucketDelete.py on line 16

CodeSmartBiks

The error is fixed and works for me. I suggest for your next issue please try to have these standalone files inside the related folders. Approving it for now. Good Job!

1jc · 2022-03-03T15:13:34Z

Made name changes per @nceres request please see the following:

#listing of files on git

C:\Users\jcdam\ad440-winter2022-thursday-repo\devops>git ls-files
DeployCloudformationS3Bucket/StackTemplate.json
DeployCloudformationS3Bucket/deployStackS3Bucket.py
DeployDynamoDB/deployDynamoDB.py
DeployDynamoDB/dynamoDBStackTemplate.json
DeployLambdaAutomation/automationScriptLambda.sh
DeployLambdaAutomation/cloudFormationLambdaTemplate.yml
DeploySagemakerNBI/deploySagemakerNBI.py
DeploySagemakerNBI/sagemakerStackTemplate.json
s3BucketDelete.py
s3BucketList.py

#changing the names if two files

C:\Users\jcdam\ad440-winter2022-thursday-repo\devops>git mv s3BucketDelete.py DeleteS3Bucket.py
C:\Users\jcdam\ad440-winter2022-thursday-repo\devops>git mv s3BucketList.py ListS3Bucket.py

#results from changing names in files

C:\Users\jcdam\ad440-winter2022-thursday-repo\devops>git ls-files
DeleteS3Bucket.py
DeployCloudformationS3Bucket/StackTemplate.json
DeployCloudformationS3Bucket/deployStackS3Bucket.py
DeployDynamoDB/deployDynamoDB.py
DeployDynamoDB/dynamoDBStackTemplate.json
DeployLambdaAutomation/automationScriptLambda.sh
DeployLambdaAutomation/cloudFormationLambdaTemplate.yml
DeploySagemakerNBI/deploySagemakerNBI.py
DeploySagemakerNBI/sagemakerStackTemplate.json
ListS3Bucket.py

#getting status of name changes

C:\Users\jcdam\ad440-winter2022-thursday-repo\devops>git status
On branch issue_#87_create_cleanup_script_for_all_s3_buckets
Your branch is up to date with 'origin/issue_#87_create_cleanup_script_for_all_s3_buckets'.

Changes to be committed:
(use "git restore --staged ..." to unstage)
renamed: s3BucketDelete.py -> DeleteS3Bucket.py
renamed: s3BucketList.py -> ListS3Bucket.py

C:\Users\jcdam\ad440-winter2022-thursday-repo\devops>git commit -m "Renamed two files by 1jc "
[issue_#87_create_cleanup_script_for_all_s3_buckets 9c9f164] Renamed two files by 1jc
2 files changed, 0 insertions(+), 0 deletions(-)
rename devops/{s3BucketDelete.py => DeleteS3Bucket.py} (100%)
rename devops/{s3BucketList.py => ListS3Bucket.py} (100%)

#committed changes

1jc · 2022-03-03T17:09:36Z

1jc made requested changes of renaming files. See above comment of changes made in git. And made requested change on line 16 error on DeleteS3Bucket.py script

nceres · 2022-03-03T23:30:46Z

@nceres Please add these scripts to their own subfolder (DeleteS3Bucket) under 'devops'. I suggest the naming convention 'deleteS3Bucket.py' to be consistent with the other folders/scripts.

I tried doing what you suggested when I first started the issue the folder system would not let me for some reason - so I put these into my own branch

@nceres Please add these scripts to their own subfolder (DeleteS3Bucket) under 'devops'. I suggest the naming convention 'deleteS3Bucket.py' to be consistent with the other folders/scripts. I tried doing what you suggested when I first started the issue the folder system would not let me for some reason - so I put these into my own branch. For some reason when they made corrections on behalf in the beginning it is where I could not make any subfolders in the devops to deter me. @CodeSmartBiks

What happens when you create a folder under devops in your editor (is it Visual Studio Code?) and try to add the files there? Is there an error message or something you can share?

1jc Issue 116 Fix script to delete all s3 buckets even with data python file DeletingAllS3BucketsIssue116.py

1jc for Issue 116 Sprint 4 " Issue 116 Fix script to delete all s3 buckets even with data"

1jc · 2022-03-15T09:05:03Z

1jc corrected sprint 3 mistake on milestone

1jc made minor correction to list buckets then delete buckets to update file "DeleteAllS33BucketIssue116.py"

…ipt_to_delete_all_s3_buckets_even_with_data DeletingAllS3BucketsIssue116.py

1jc · 2022-03-22T14:37:31Z

1jc moved to Done 03222022

Changed to Sprint 3

1jc added 2 commits March 1, 2022 11:22

Create s3BucketList.py

e814b63

1jc created script to list s3 bucket listing

Create s3BucketDelete.py

3e14cf7

1jc created script to delete a s3 bucket listing for clean up

1jc self-assigned this Mar 1, 2022

1jc added the DevOps Team label Mar 1, 2022

1jc added this to the Sprint 04 milestone Mar 1, 2022

1jc linked an issue Mar 1, 2022 that may be closed by this pull request

Create a cleanup script for all S3 buckets #87

Open

1jc requested review from dagmawidemissie17 and CodeSmartBiks March 1, 2022 23:00

1jc added 2 commits March 2, 2022 21:45

Update s3BucketDelete.py

e439b88

1jc made changes to line 16 per request BT

Create s3BucketDelete.py

68fe4d7

1jc made changes to line 16 per request BT

nceres requested changes Mar 3, 2022

View reviewed changes

1jc requested a review from nceres March 3, 2022 06:30

CodeSmartBiks approved these changes Mar 3, 2022

View reviewed changes

1jc added 3 commits March 14, 2022 16:21

Create DeletingAllS3BucketsIssue116

5d76876

1jc Issue 116 Fix script to delete all s3 buckets even with data python file DeletingAllS3BucketsIssue116.py

Delete DeletingAllS3BucketsIssue116

6a1a2a3

DeletingAllS3BucketsIssue116.py

4eb4291

1jc for Issue 116 Sprint 4 " Issue 116 Fix script to delete all s3 buckets even with data"

1jc modified the milestones: Sprint 04, Sprint 03 Mar 15, 2022

1jc added 2 commits March 16, 2022 21:16

Update DeleteAllS3BucketsIssue116.py

a8cb2b5

1jc made minor correction to list buckets then delete buckets to update file "DeleteAllS33BucketIssue116.py"

Merge pull request #139 from North-Seattle-College/Issue_#116_Fix_scr…

7ec739e

…ipt_to_delete_all_s3_buckets_even_with_data DeletingAllS3BucketsIssue116.py

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue #87 create cleanup script for all s3 buckets #109

Issue #87 create cleanup script for all s3 buckets #109

1jc commented Mar 1, 2022 •

edited

Loading

1jc commented Mar 1, 2022 •

edited

Loading

1jc commented Mar 1, 2022

1jc commented Mar 1, 2022

1jc commented Mar 1, 2022

1jc commented Mar 1, 2022

nceres commented Mar 3, 2022

nceres commented Mar 3, 2022

nceres left a comment

1jc commented Mar 3, 2022

1jc commented Mar 3, 2022

CodeSmartBiks left a comment

1jc commented Mar 3, 2022

1jc commented Mar 3, 2022

nceres commented Mar 3, 2022

1jc commented Mar 15, 2022

1jc commented Mar 22, 2022

Issue #87 create cleanup script for all s3 buckets #109

Are you sure you want to change the base?

Issue #87 create cleanup script for all s3 buckets #109

Conversation

1jc commented Mar 1, 2022 • edited Loading

1jc commented Mar 1, 2022 • edited Loading

1jc commented Mar 1, 2022

1jc commented Mar 1, 2022

1jc commented Mar 1, 2022

1jc commented Mar 1, 2022

nceres commented Mar 3, 2022

nceres commented Mar 3, 2022

nceres left a comment

Choose a reason for hiding this comment

1jc commented Mar 3, 2022

1jc commented Mar 3, 2022

CodeSmartBiks left a comment

Choose a reason for hiding this comment

1jc commented Mar 3, 2022

1jc commented Mar 3, 2022

nceres commented Mar 3, 2022

1jc commented Mar 15, 2022

1jc commented Mar 22, 2022

1jc commented Mar 1, 2022 •

edited

Loading

1jc commented Mar 1, 2022 •

edited

Loading